Skip to content

Ditto MCP server — knowledge RAG + credential-passthrough action tools - #2522

Open
kalinkostashki wants to merge 11 commits into
eclipse-ditto:masterfrom
boschglobal:ditto-mcp-server
Open

Ditto MCP server — knowledge RAG + credential-passthrough action tools#2522
kalinkostashki wants to merge 11 commits into
eclipse-ditto:masterfrom
boschglobal:ditto-mcp-server

Conversation

@kalinkostashki

@kalinkostashki kalinkostashki commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds an extensible Ditto MCP server (TypeScript, @modelcontextprotocol/sdk) under mcp/. It gives AI agents (e.g. Claude Code) two capabilities over one server:

  1. Knowledge / RAG — searchable Ditto documentation (public llms.txt + optional local corpus) via keyword, semantic, or hybrid retrieval.
  2. Ditto action tools — auto-generated from the OpenAPI spec, with credential passthrough (the MCP forwards the caller's credentials; Ditto authorizes) and a ToolPolicy (read-only by default; writes and sudo/devops are opt-in).

All new code lives in mcp/; no existing files are modified. 31 test files; default suite is hermetic (Postgres tests are opt-in behind test:pg + Docker).

What's included (by commit / phase)

  • Foundation — Zod-validated config + loader, core ToolDef/ToolRegistry, ping, server factory, stdio and streamable-HTTP transports (loopback bind, session store, DNS-rebinding protection), request-context threading.
  • Knowledge core (FTS) — chunker, SQLite FTS5 retriever, PublicSource (llms.txt), KnowledgeService + search/get_chunk tools, one shared service across sessions.
  • Semantic + hybrid retrieval — local ONNX bge-small embeddings, sqlite-vec store, HybridRetriever (reciprocal-rank fusion), local-dir corpus, config-selectable fts/vector/hybrid, lazy ONNX load, markdown-only ingest, bounded-concurrency fetch, batched embeddings (bounds memory), retrieval provenance.
  • Persistent knowledge index — file-backed KnowledgeStore (chunks + FTS + vectors), ingest CLI, atomic ingest (temp + rename), index-metadata validation, async store lifecycle + openStore factory, PgKnowledgeStore (pgvector + Postgres tsvector) with testcontainers tests.
  • Ditto action tools + auth — OpenAPI parser + operation-to-tool generation with a pinned bundled spec fallback, HttpDittoClient, credential passthrough (basic / devops / OIDC client-credentials; config-level and per-session Authorization header), ToolPolicy (allow-methods + write allowlist + spec-security-aware sudo gating for /devops, /sudo, /connections), shallow typed request-body schemas.
  • Documentation + OSS packaging — consolidated mcp/README.md, five example configs, hardened .gitignore.

Configuration scenarios (see mcp/examples/)

  • public-fts.json — OSS default: public docs, keyword search, no Ditto.
  • hybrid-local.json — hybrid RAG over public docs + a local corpus.
  • pgvector.json — shared Postgres/pgvector index.
  • ditto-readonly.json — docs + read-only (GET) Ditto tools.
  • ditto-oidc-write.json — OIDC creds + selected write tools.

Testing

Verified end-to-end against a live k3d Ditto (basic ditto:ditto, devops devops:devops):

  • Ingested the full public corpus → 1033 chunks (hybrid, bge, metadata complete).
  • tools/list under a GET-only policy exposed zero write/sudo/connection tools (policy filtering works).
  • Semantic search returned relevant MQTT/reconnect docs (FTS + vector both firing).
  • Live Ditto GET via an MCP tool → HTTP 200.
  • Allow-listed PUT (create thing) → HTTP 201, confirmed independently.
  • Sudo /connections with a non-devops credential → refused; with a devops credential → gate passes, Ditto returns 200.
  • Passthrough over HTTP transport, driven by Claude Code as the MCP client: server configured with no stored credentials, Authorization header supplied per request → thing list/search reached Ditto using the forwarded credentials only.

Security notes

  • The MCP performs no authorization itself — it forwards credentials and lets Ditto decide.
  • Read-only is the default; writes and sudo/devops operations are opt-in via explicit allowlists.
  • Credentials are never logged. Internal corpora are protected by deployment isolation, not by MCP-side auth.
  • HTTP transport binds loopback by default with DNS-rebinding protection on.

How to run

cd mcp && npm install && npm run build
DITTO_MCP_CONFIG=examples/public-fts.json node dist/bin/ingest.js # build the index
DITTO_MCP_CONFIG=examples/public-fts.json node dist/bin/stdio.js # or dist/bin/http.js

kalinkostashki and others added 10 commits August 10, 2026 14:32
…TTP transports

Zod-validated AppConfig + loader, ToolDef/ToolRegistry core, ping tool, server factory, stdio and streamable-HTTP entrypoints (loopback bind, session store, DNS-rebinding protection), request-context threading.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Chunk/KnowledgeSource/Retriever types, markdown chunker, SQLite FTS5 retriever, PublicSource llms.txt loader, KnowledgeService + search/get_chunk tools wired via config, shared async service across sessions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…RRF fusion

Local ONNX bge-small embeddings, sqlite-vec store + VectorRetriever, HybridRetriever (reciprocal rank fusion), LocalDirSource corpus, config-selectable fts/vector/hybrid, lazy ONNX load, markdown-only ingest, bounded-concurrency fetch, batched embeddings (OOM fix), limit param + provenance.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…, pgvector

File-backed KnowledgeStore (chunks+FTS5+vectors), read-only retrievers over the store, ingest CLI, atomic ingest (temp+rename), index metadata validation, async store lifecycle + openStore factory, PgKnowledgeStore (pgvector + tsvector) with testcontainers pg tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ToolPolicy

OpenAPI operation parser + operation-to-tool generation with pinned bundled spec, HttpDittoClient, credential passthrough (basic/devops/OIDC client-credentials, config + per-session), ToolPolicy (read-only default, write allowlist, spec-security-aware sudo gating), typed request-body schemas.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Consolidated README, example configs, hardened .gitignore; docs for HTTP/config, knowledge/persistence/pgvector, action tools/passthrough/policy, OIDC + typed bodies.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Squashes the devops-credential and openapi-spec-source work.

Devops credential:
- Add optional `ditto.devopsCredential` used exclusively for
  sudo-classified operations (/devops/*, sudo*, and the secret-bearing
  /api/2/connections* API); all other operations use `ditto.credential`.
- Sudo ops are refused at the MCP layer when no devopsCredential is set.
  Connectivity is classified sudo via a spec-independent path rule so it
  is always devops-gated.
- Remove the `devops` credential kind and the `isDevops` flag: devops
  capability is now positional (which credential slot). A per-session
  Authorization header overrides the selected credential without
  inheriting devops status. Credential kind is now `basic | oidc` only.

OpenAPI spec source:
- Resolve the spec by precedence path > url > version > in-repo canonical;
  add `openApi.version` (git tag/ref) and `openApi.versionUrlTemplate`.
- Default fallback reads the canonical in-repo spec
  (documentation/src/main/resources/openapi/ditto-api-2.yml) instead of a
  committed duplicate; drop mcp/assets/ditto-openapi.yml.

BREAKING CHANGE: kind:"devops" and the devops:true flag are removed; move a
devops credential into ditto.devopsCredential. Sudo operations now require
ditto.devopsCredential to be set.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Dockerfile (multistage node:22-slim, glibc), .dockerignore, and
  docker/config.docker.json: HTTP transport, non-root, native deps baked in.
  Secure-by-default DNS rebinding protection + loopback allowedHosts.
- bin shebangs so stdio/http/ingest run as installed CLIs.
- Lazy-load sqlite/pg store backends and local embeddings; friendly errors
  naming the missing native dep instead of raw MODULE_NOT_FOUND.
- knowledge.chunk.{maxChars,overlap} config wired through makeSources to
  PublicSource + LocalDirSource (was hardcoded to chunker defaults).
- README: Docker Deployment and Chunking sections (incl. bge-small
  512-token ceiling for vector/hybrid).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Retriever-agnostic context expansion in KnowledgeService.search: around each
anchor, pull ±context same-document chunks (same cite, adjacent ordinal) by id
— never re-scored or re-embedded. Results are deduped and emitted as contiguous
spans ordered by best anchor rank; neighbors are tagged role="context" so their
relevance isn't over-weighted vs. matches.

- Stops at document boundaries (different cite) even when ordinals are globally
  contiguous, so expansion never leaks across docs.
- knowledge.search.{limit,context} config (defaults 5 / 1); search tool gains a
  per-call `context` arg that overrides. `limit` counts anchors; neighbors extra.
- Works for fts/vector/hybrid: an anchor's neighbors are defined by document
  layout, not by how it was matched — so expansion is embedding-neutral and
  sidesteps the 512-token ceiling.
- README: Search (query-time) section incl. overlap-vs-expansion interaction.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kalinkostashki kalinkostashki changed the title Draft: Ditto MCP server — knowledge RAG + credential-passthrough action tools Ditto MCP server — knowledge RAG + credential-passthrough action tools Aug 18, 2026
- removed writeAllowlist underscores from configs
- stripped unnecessary underscores from tools generation
- updated documentation to reflect the changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants